Variable Importance using Decision Trees, Supplementary Material for NIPS 2017 paper

We present a complete presentation of the theoretical results presented in the main text. We provide detailed analysis of the DStump algorithm in the context of a general additive regression model with uncorrelated design. We derive the results for the linear case as special case of the general theory. Our analysis is high-dimensional and non-asymptotic, and to our knowledge the first such analysis of feature selection properties of decision trees. We show that even in the high-dimensional setting where the number of features p is much larger that the sample size n, feature importance scores based on impurity reduction contain enough information for consistent model selection. Additionally, we provide simulation experiments to supplement the results in the main text.